Practical writing experience is generally regarded as an effective method of developing writing skills. In this regard, literature pertaining to the teaching of writing suggests that evaluation and feedback, specifically pointing out strong and weak areas in a student's essay writing, may facilitate improvements in the student's writing abilities. This is particularly so with regard to essay coherence.
In traditional writing classes, an instructor evaluates student essays. This evaluation typically includes comments directed to specific elements of the essay. Similarly, with the advent of automated essay evaluation, a computer application can be configured to evaluate an essay and provide feedback. For particular writing errors, such as misspellings or subject-verb agreement, this process is straightforward. Word spellings, for instance, can be compared against a list of correctly spelled words. Any words not found in the list are determined to be incorrectly spelled. Errors in subject-verb agreement can be identified based on a corpus of annotated essays.
In contrast, providing feedback regarding a student's writing style is typically more subjective in nature. Essay coherence, where the correlation between segments of text is evaluated, is one area where subjective feedback is present. Cohesion relates to both the correlation of the text to various smaller segments within the text and relatedness of the text to a test prompt.
Essay coherence is related to semantic similarity between various segments of text. For example, words within sentences of a discussion section of an essay should be similar to the words used in a topic sentence for the discussion section in order for the essay to be coherent.
Determining whether two sentences in a text are related depends on many factors, such as whether the two sentences refer to the same entities, whether they relate to the same topic, and whether anaphoric connections exist between them. Given the current state of natural language understanding, a determination of whether two text segments are related requires a representation of the similarity of their semantic content. A semantic similarity metric based on the relatedness of sentences within an essay can be devised because related sentences in a text typically tend to use the same or similar words. Thus, the use of similar words functions as a predictor of relatedness.
One known method of evaluating an essay for coherence includes the use of a vector-based similarity calculation between text segments to measure relatedness. In such a method, vectors represent text segments after modeling. The cosine between each pair of adjacent sentences is then calculated. A very high cosine value would indicate excessive redundancy between the sentences. A low cosine value indicates a conceptual shift in the text. A conceptual shift can occur if the text is not well connected or if the essay writer switches to a new topic. A cohesive essay should have neither high nor low cosine values. The method can respond to these scenarios and provide the appropriate feedback to the student. By taking the average cosine between adjacent sentences, in a linear manner, a measure of the overall coherence of the text is evaluated.
Latent Semantic Analysis (LSA) is a method of representing text segments as vectors. LSA is a statistical model of word usage that compares the semantic similarity between text segments. In order to analyze an essay, LSA first generates a matrix of occurrences of each word in each document (sentences or paragraphs). LSA then uses singular-value decomposition (SVD), a technique closely related to eigenvector decomposition and factor analysis. The SVD scaling decomposes the word-by-document matrix into a set of k, typically 100 to 300, orthogonal factors from which the original matrix can be approximated by linear combination. Instead of representing documents and terms directly as vectors of independent words, LSA represents them as continuous values on each of the k orthogonal indexing dimensions derived from the SVD analysis. Since the number of factors or dimensions is much smaller than the number of unique terms, words will not be independent. For example, if two terms are used in similar contexts (documents), the terms will have similar vectors in the reduced-dimensional LSA representation.
One can interpret the analysis performed by SVD geometrically. The result of the SVD is a k-dimensional vector space containing a vector for each term and each document. The location of term vectors reflects the correlations in their usage across documents. Similarly, the location of document vectors reflects correlations in the terms used in the documents. In this space, the cosine or dot product between vectors corresponds to their estimated semantic similarity. Thus, by determining the vectors of two pieces of textual information, the semantic similarity between them can be determined.
In LSA, the rows (and the columns) of the frequency matrix can be interpreted as multi-dimensional context vectors where the elements are normalized frequency counts and the dimensionality is the number of contexts in the text data. Thus, the representations are local. The inherent problem with using local representations in natural language processing is that the size, or dimensionality, of the representations grows with the size of the data. This means that the model does not scale and that the co-occurrence matrix can become computationally intractable as the vocabulary and the document collection grow. In contrast, reducing the dimensionality of the matrix can make the method computationally feasible.
In LSA, a vector for a new document is obtained by making a sparse vector of the length of the vocabulary, indicating the frequency of each term in the document, and multiplying this vector by the term matrix T, in order to map the vector to the reduced space. The vector representation for a text segment (for example, a sentence) equals the vector sum of the term vectors for each word in the segment. The term vectors could have previously been normalized to unit length, and a stoplist can be used to prevent the vectors for function words from being included in the sum.
Several drawbacks exist for the known systems of text coherence evaluation which simply calculate the similarity between adjacent sentences in a text, assuming that the chain of text coherence is essentially linear.
Accordingly, it is necessary to develop a method of evaluating the coherence of an essay including various discourse elements and a plurality of text segments, wherein the essay is annotated, the text segments are represented by vectors, and the text segment vectors are compared in a more coherent manner than a simple linear comparison.
Additionally, it is necessary to develop a method of representing text segments as vectors that overcomes the limitations of previous technologies, such as LSA and calculating cosines between adjacent sentences.
Finally, it is necessary to develop a system that generates quantitative and/or qualitative feedback on discourse elements and essays.
The present invention is directed towards solving one or more of these problems.